Most systems, whether it is a hardware system or a software system, requires quality assurance (QA) and system verification tests (SVT) before it is released for actual use by the public. Conventional methods of SVT include convention manual SVT and conventional automated SVT.
The conventional manual SVT process typically involves the steps of (i) gaining an understanding of the feature to be tested on the SUT, (ii) designing one or more test cases to validate the features, (iii) executing the test case(s) manually, and (iv) analyzing the results of the executed test case(s). Obviously, the goal is to verify that the SUT meets the requirements for how a given feature is intended to operate based on the execution results of the test case(s).
The disadvantage of conventional manual SVT is that it requires regression testing to be done manually, i.e., manually repeating the test cases when the SUT has changed, typically because of a new build, new release, or changes in the SUT's hardware or software. Regression testing using conventional manual SVT is labor-intensive, time-consuming, and error-prone, because there is no automation involved.
Conventional automated SVT replaces the labor involved in repetitive conventional manual SVT with work that is performed automatically by a computer. The conventional automated SVT process typically involves (i) gaining an understanding of the feature to be tested on the SUT, (ii) designing one or more test cases to validate the feature, (iii) executing the test case(s) manually, (iv) analyzing the results of the executed test case, (v) designing a script that can execute the test case(s) automatically, (vi) testing the script to make sure that it produces no false-negatives and no false-positives, (vii) configuring an automation infrastructure so that the script will run at regular intervals automatically, (viii) reviewing the results from automated regression testing and identifying test cases that require review, (ix) analyzing the results from failed scripts, i.e., filing defect reports and annotating them accordingly, and (x) modifying the scripts as necessary when the script has reported a false-negative failure, typically because the SUT's behavior has changed in an acceptable way that was not anticipated by the script.
A script is procedural, i.e., it describes a series of steps to follow, some of which invoke certain actions to be taken on the SUT. Other steps are to check certain reactions from the SUT to the actions. Procedural computer programming languages, such as C, TCL, PERL, etc. are typically used to design the scripts used in conventional automated SVT.
The benefit of conventional automated SVT is that the test case is automatically executed repeatedly and that test cases and failed scripts are further reviewed and modified, if necessary, in steps (viii)-(x) above. However, because the human labor involved in designing a script that can execute a test case, testing the script to make sure that it produces no false-negatives and no false-positives, and modifying the scripts as necessary when the script has reported a false-negative failure are significant, the benefit from test automation comes only after the test case is executed many times. A lot of times, intense human labor is required to design a script for executing a test case.
Therefore, there is a need for an automated SVT process that can eliminate the repetitive, mundane, labor-intensive aspects of designing, testing, and maintaining test automation scripts for executing test cases. There is also a need for an automated SVT process that can eliminate the use of scripts for executing test cases.